The present invention, in some embodiments thereof, relates to method and system of communication management and/or monitoring and, more particularly, but not exclusively, to method and system of managing and/or monitoring communication among a plurality of nodes.
As the scale, the volume and the rate of the data streams of today's networked systems continue to increase, algorithms and tools for effectively analyzing them are becoming an important research mandate. Powerful tools for sophisticated data processing and analysis such as relational and online analytical processing (OLAP) query engines, decision trees, association rule mining, feature selection and classification, ranking, statistical analysis, phase-change detection, anomaly and outlier detection, and the like are used for extracting correlations, patterns, and knowledge. These tools are used for understanding hidden mechanisms, decision making, optimizing the operation of complex and autonomous systems, improving user interaction with automated services, and many other critical tasks. For example, in a keynote he gave on “Extreme Data mining” at SIGMOD 2008, Sridhar Ramaswamy from Google said, referring to the log files generated at the Google data centre: “they are our only source of truth”. At the same time they are “orders of magnitude larger than any other data in our system”.
The data-management and data-mining research communities pride themselves on the ability of their technologies to scale to large amounts of data. Traditional data processing and analysis systems are built on the assumption of centralized and static data sets, an assumption which is not valid in large-scale networked systems in which a plurality of connected, distributed, dynamic sources of evolving information streams, are used. Two crucial scalability dimensions which effect the computation in distributed networked systems are the degree of distribution and the dynamicity of the data. The degree of distribution is a key scalability metric for massive networked systems, having a plurality of nodes, for example hundreds of millions, such as the Internet and P2P networks. In addition, the information residing in such systems is invariably dynamic in nature, with continuous streams of data flowing from a variety of sources, including packet traces, system logs, communication protocols, multiple sensor modalities, and mobile pervasive-computing devices and applications.
An example for such distributed systems includes a monitoring system having a plurality of distributed sensors. For example, a tsunami detection system that has a plurality of distributed sensors for measuring water pressure over a wide area in the ocean. The combined readings of these sensors are used to predict whether a tsunami is going to strike. The system must be able to issue correct warnings to save lives; however, false alarms are undesirable, as they may result in the needless measures, such as evacuation. Power and computational resource limitations define the continuity of the sensors' broadcast of their readings to a central monitoring station. Wireless sensors have restricted power supply that can support a limited number of transmissions and the central node usually cannot analyze all of the incoming data on the fly. In other distributed domains, more restrictions come to play, such as huge data volumes: for example, in a network-traffic monitoring scenario, a simple calculation shows that capturing only IP-packet-header information over a single 1 Gigabit/s Ethernet link can easily give rise to Terabytes of data each day.
During the last years, various solutions have been developed to improve and decrease the reaction time of distributed systems. For example, U.S. Patent Application No. 2009/0310496, filed on Mar. 14, 2007 describes a method for distributed computing, which includes processing multiple sets of data at respective computing nodes, and calculating respective local values of one or more statistical parameters characterizing the sets of the data. A global condition is defined, such that the condition is violated when a function defined over a weighted average of the respective local values crosses a predetermined threshold. The global condition is separated into a plurality of local geometric constraints, which include a respective local geometric constraint to be evaluated by each of the nodes based on the respective local values, such that violation of the respective local geometric constraint in at least one of the nodes indicates a violation of the global condition. The local geometric constraint is evaluated independently at each of the nodes. When at least one of the nodes detects that the respective local geometric constraint is violated, an indication that the global condition has been violated is produced.